[Tooling] Add google owned cache for dependencies as an option in ci #4567

reidbaker · 2023-07-26T15:49:34Z

Proof of concept of a package using artifact hub.
Artifact hub is a google owned and managed cache of google/maven dependencies.
Using this cache should decrease the number of flakes related to downloading dependencies along with future benefits of licence analysis and security alerts.

Read more at go/artifact-hub#maven
flutter/flutter/issues/120119

Next steps:
Enable a new env variable on CI servers. https://flutter-review.googlesource.com/c/recipes/+/48260 cl/551888350
Merge this PR.
Apply this change to all packages
Add enforcement as part of the gradle check.

Developers have a new dependency for buildscript but it is publicly available, so I added a next changelog.

Pre-launch Checklist

Requried to be an authenticated googler `gcloud auth application-default login` Useful links https://g3doc.corp.google.com/company/teams/artifact-hub/index.md?cl=head#maven https://github.com/GoogleCloudPlatform/artifact-registry-maven-tools/blob/master/README.md Useful command ``` gcloud artifacts print-settings mvn \ --project=artifact-foundry-prod \ --repository=maven-3p \ --location=us ```

…epository addition

flutter-dashboard · 2023-07-26T15:49:37Z

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

packages/camera/camera_android_camerax/example/android/build.gradle

GRADLE_STRUCTURE.md

packages/camera/camera_android_camerax/example/android/build.gradle

gmackall

LGTM!

GRADLE_STRUCTURE.md

stuartmorgan-g · 2023-07-28T14:32:48Z

GRADLE_STRUCTURE.md

@@ -0,0 +1,16 @@
+# Gradle Structure


Having seen the details here, I think a wiki page referenced from https://github.com/flutter/flutter/wiki/Understanding-Packages-tests#external-build-dependencies and maybe https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure would probably be better than an md file.

ok in that case consider the md files content for review and I will remove and update the wiki before merging.

stuartmorgan-g · 2023-07-28T14:32:57Z

GRADLE_STRUCTURE.md

@@ -0,0 +1,16 @@
+# Gradle Structure
+
+`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.


stuartmorgan-g · 2023-07-28T14:33:19Z

GRADLE_STRUCTURE.md

+
+`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.
+
+This repo has a GCP instance that mirrors dependencies available from `google()` and `mavenCentral()` used by ci (or googlers). This gives us redundant uptime for dependency availability.


Nits: CI, Googlers

stuartmorgan-g · 2023-07-28T14:33:45Z

GRADLE_STRUCTURE.md

+
+Using this cache is not intended or avaiable for contributors outside of CI. We protect that execution with an environment variable `ARTIFACT_HUB_REPOSITORY` to ensure that by default users do not see rejected cloud credentials or errors in builds.
+
+Googlers can debug locally by setting `ARTIFACT_HUB_REPOSITORY` to the valid artifact hub value and authenticating with GCP. To authenticate run `gcloud auth application-default login`. To find artifact hub url use `<url>` section of go/artifact-hub#maven or inspect the value on CI servers. CI uses a service account for billing. That is defined in go/artifact-hub-service-account (googler access only).


Nit: Googler access

stuartmorgan-g · 2023-07-28T14:34:38Z

packages/camera/camera_android_camerax/CHANGELOG.md

@@ -1,3 +1,7 @@
+## NEXT
+
+* Updates android example to be able to use custom use artifactregistery.


Since nobody outside the Flutter team can use this, let's not put it in the CHANGELOG.

They can if they setup a artifactrepository then setup the env variables on their local machine (I just updated the file to reflect this). I removed the google specific access parts to configuration. Also mechanically there is a new dependency and the changelog thing scolded me.

I can remove this if you want even though there is technically a developer facing change that should not impact the.

removed and replaced with https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure#gradle-structure

stuartmorgan-g · 2023-07-28T14:36:13Z

Overriding changelog check: see #4567 (comment)

reidbaker · 2023-07-28T15:07:32Z

Approved by artifact hub team via email by sfelix@

Replaced with https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure#gradle-structure

reidbaker · 2023-08-01T14:39:37Z

Confirmed the environment variable was set correctly https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/execution_details and confirmed the artifact hub variable was used
Using artifact hub
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/stdout

reidbaker · 2023-08-01T16:14:55Z

Confirmed the environment variable was set correctly https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/execution_details and confirmed the artifact hub variable was used Using artifact hub https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8773957271362564241/+/u/Run_package_tests/download_Dart_and_Android_deps/stdout

https://critique.corp.google.com/cl/552818879 had not landed. Links above show our configuration is correct and that if we lose access to artifact hub our builds will not go red. I do not know of a way to verify we are getting the dependencies from the artifact hub server without ssh access to the machine while they are being downloaded.

In a future pr I could make ci only use the artifact hub instance but I am not confident enough yet to make that call.

…n in ci (flutter/packages#4567)

flutter/packages@4e4961a...d00c1f9 2023-08-03 49699333+dependabot[bot]@users.noreply.github.com Bump github/codeql-action from 2.21.1 to 2.21.2 (flutter/packages#4637) 2023-08-03 [email protected] Roll Flutter from b3f99ff to c00d241 (12 revisions) (flutter/packages#4640) 2023-08-03 [email protected] [path_provider] Add getApplicationCachePath() - implementations (flutter/packages#4619) 2023-08-03 [email protected] [various] Removed references to deprecated `TestWindow` APIs (flutter/packages#4558) 2023-08-02 49699333+dependabot[bot]@users.noreply.github.com Bump actions/labeler from 4.1.0 to 4.3.0 (flutter/packages#4432) 2023-08-02 [email protected] [go_router_builder] Add go_router StatefulShellRoute support to go_router_builder (flutter/packages#4238) 2023-08-02 [email protected] [Tooling] Add google owned cache for dependencies as an option in ci (flutter/packages#4567) 2023-08-02 [email protected] Roll Flutter from 1d59196 to b3f99ff (32 revisions) (flutter/packages#4634) 2023-08-02 [email protected] [go_router_builder] Support `ShellRouteData` without `const` constructor (flutter/packages#4627) 2023-08-02 49699333+dependabot[bot]@users.noreply.github.com Bump github/codeql-action from 2.21.0 to 2.21.1 (flutter/packages#4573) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

- Adds artifact hub check to gradle command - Add tests for build.gradle and settings.gradle check - Update all example build.gradle and settings.gradle files flutter/flutter/issues/120119 Expansion of #4567

- Adds artifact hub check to gradle command - Add tests for build.gradle and settings.gradle check - Update all example build.gradle and settings.gradle files flutter/flutter/issues/120119 Expansion of flutter/packages#4567

Service account setup cl/551888350 See flutter/packages#4567 See https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure#gradle-structure Bug: flutter/flutter/issues/120119 Change-Id: Ic1c4dd7c348854b21f9dedd02d664edc34e9bcec Reviewed-on: https://flutter-review.googlesource.com/c/recipes/+/48260 Commit-Queue: Reid Baker <[email protected]> Reviewed-by: Keyong Han <[email protected]>

- Adds artifact hub check to gradle command - Add tests for build.gradle and settings.gradle check - Update all example build.gradle and settings.gradle files flutter/flutter/issues/120119 Expansion of flutter/packages#4567

reidbaker added 2 commits July 25, 2023 18:18

Update documentation, split out plugin apply into setting, simplify r…

befa254

…epository addition

reidbaker requested a review from camsim99 as a code owner July 26, 2023 15:49

github-actions bot added p: camera platform-android labels Jul 26, 2023

reidbaker changed the title y [Tooling] Add google owned cache for dependencies as an option in ci Jul 26, 2023

reidbaker requested review from stuartmorgan-g, johnmccutchan and godofredoc July 26, 2023 15:55

johnmccutchan approved these changes Jul 26, 2023

View reviewed changes

johnmccutchan reviewed Jul 26, 2023

View reviewed changes

packages/camera/camera_android_camerax/example/android/build.gradle Outdated Show resolved Hide resolved

Update the env variable to be more specific

73aea86

gmackall reviewed Jul 26, 2023

View reviewed changes

GRADLE_STRUCTURE.md Outdated Show resolved Hide resolved

GRADLE_STRUCTURE.md Outdated Show resolved Hide resolved

packages/camera/camera_android_camerax/example/android/build.gradle Outdated Show resolved Hide resolved

Speling and phrasing updates, make key a variable

a2e87fb

gmackall approved these changes Jul 26, 2023

View reviewed changes

gmackall reviewed Jul 26, 2023

View reviewed changes

GRADLE_STRUCTURE.md Outdated Show resolved Hide resolved

reidbaker added 4 commits July 26, 2023 16:02

formatting

474f547

Merge branch 'main' into i120119-artifact-hub-poc

4747b49

Update changelog

797e066

Merge branch 'main' into i120119-artifact-hub-poc

f249971

stuartmorgan-g reviewed Jul 28, 2023

View reviewed changes

stuartmorgan-g added the override: no changelog needed Override the check requiring CHANGELOG updates for most changes label Jul 28, 2023

reidbaker added 4 commits July 28, 2023 10:58

Update GRADLE_STRUCTURE.md to use CI instead of ci

81b0406

Update GRADLE_STRUCTURE.md

a996c06

Merge branch 'main' into i120119-artifact-hub-poc

2adfc43

Update CHANGELOG.md to not have comment

3ce0495

Delete GRADLE_STRUCTURE.md

6fc5753

Replaced with https://github.com/flutter/flutter/wiki/Plugins-and-Packages-repository-structure#gradle-structure

reidbaker added 2 commits July 28, 2023 12:36

Update settings.gradle

621aee2

Merge branch 'main' into i120119-artifact-hub-poc

875b078

stuartmorgan-g approved these changes Jul 31, 2023

View reviewed changes

Merge branch 'main' into i120119-artifact-hub-poc

77266fd

reidbaker added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 1, 2023

Merge branch 'main' into i120119-artifact-hub-poc

23e9840

auto-submit bot merged commit 11b79b5 into flutter:main Aug 2, 2023

reidbaker deleted the i120119-artifact-hub-poc branch August 3, 2023 14:15

engine-flutter-autoroll mentioned this pull request Aug 3, 2023

Roll Packages from 4e4961a24b3d to 9323e33ed9d8 (8 revisions) flutter/flutter#131852

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Aug 3, 2023

11b79b5 [Tooling] Add google owned cache for dependencies as an optio…

2fb25d4

…n in ci (flutter/packages#4567)

engine-flutter-autoroll mentioned this pull request Aug 3, 2023

Roll Packages from 4e4961a24b3d to 94ec57fe2583 (9 revisions) flutter/flutter#131855

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Aug 3, 2023

11b79b5 [Tooling] Add google owned cache for dependencies as an optio…

4975804

…n in ci (flutter/packages#4567)

engine-flutter-autoroll mentioned this pull request Aug 3, 2023

Roll Packages from 4e4961a24b3d to d00c1f9eb01c (10 revisions) flutter/flutter#131866

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Aug 3, 2023

11b79b5 [Tooling] Add google owned cache for dependencies as an optio…

e904eb6

…n in ci (flutter/packages#4567)

reidbaker mentioned this pull request Aug 4, 2023

[All] Expand artifact hub to all plugins #4645

Merged

11 tasks

gmackall mentioned this pull request Oct 12, 2023

Mac module_custom_host_app_name_test is 2.02% flaky flutter/flutter#136327

Closed

jmagman mentioned this pull request Jan 7, 2025

Android engine test flaking on failure to fetch dependencies flutter/flutter#120119

Open

		@@ -0,0 +1,16 @@
		# Gradle Structure

		`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.


		`package/example/android/settings.gradle` imports the flutter tooling, includes the app directory, and configures GoogleCloudPlatform/artifact-registry-maven-tools for use in ci.

		This repo has a GCP instance that mirrors dependencies available from `google()` and `mavenCentral()` used by ci (or googlers). This gives us redundant uptime for dependency availability.


		Using this cache is not intended or avaiable for contributors outside of CI. We protect that execution with an environment variable `ARTIFACT_HUB_REPOSITORY` to ensure that by default users do not see rejected cloud credentials or errors in builds.

		Googlers can debug locally by setting `ARTIFACT_HUB_REPOSITORY` to the valid artifact hub value and authenticating with GCP. To authenticate run `gcloud auth application-default login`. To find artifact hub url use `<url>` section of go/artifact-hub#maven or inspect the value on CI servers. CI uses a service account for billing. That is defined in go/artifact-hub-service-account (googler access only).

		@@ -1,3 +1,7 @@
		## NEXT

		* Updates android example to be able to use custom use artifactregistery.

[Tooling] Add google owned cache for dependencies as an option in ci #4567

[Tooling] Add google owned cache for dependencies as an option in ci #4567

Uh oh!

Conversation

reidbaker commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

flutter-dashboard bot commented Jul 26, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmackall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g commented Jul 28, 2023

Uh oh!

reidbaker commented Jul 28, 2023

Uh oh!

reidbaker commented Aug 1, 2023

Uh oh!

reidbaker commented Aug 1, 2023

Uh oh!

Uh oh!

reidbaker commented Jul 26, 2023 •

edited

Loading